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ABSTRACT 

This paper describes the approach of the University of Texas 
at Arlington (UTA) to ensuring that all students have basic competence in 
computer use. It describes the development and implementation of the UTA 
computer proficiency examination. UTA students currently have two options. 
They may take the computer proficiency course designated in their major 
department or they may take the UTA Computer Proficiency Examination. The 
test is a performance-based, or authentic, assessment that is a modified 
combination of product-oriented and portfolio assessment. Rather than 
measuring a student's ability to memorize keystrokes and pulldown menus, it 
is a measure of a student's ability to use the tools available in order to 
perform meaningful, real-life tasks, although in a proctored environment that 
makes it an abbreviated version of a portfolio. The five test components are: 
word processing, spreadsheets, Internet research and evaluation, e-mail, and 
online library use. A student who fails a component may take it again 6 weeks 
later. A small student survey (n=22) has indicated that students generally 
felt prepared for the test and found that it tested content they expected. 
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Introduction 



Virtually every organization, in both the private and public sectors, has experienced a rapid 
increase in the use of technology over the past 10 years. In keeping with these changes, it has 
been necessary to adapt curriculum at every level to ensure that students have the skills 
necessary to compete in the new knowledge economy. 

While we know that students need to be prepared to use technology, identifying the skills 
that qualify as general proficiency is a difficult task. Technology is becoming increasingly 
specialized. Nearly every discipline has a software program unique to their work. On most 
campuses, the Engineering Department uses AutoCAD, the Music department uses music 
notation and sequencing software (MIDI), the Psychology Department uses SPSS, and some 
departments use nothing beyond e-mail and word processing. 

However, all students need to be knowledgeable consumers of information and be able to 
use the tools that are necessary for performance in any environment. This paper will focus on the 
incorporation of these needs into higher education graduation requirements, how a computer 
proficiency requirement can be implemented, and the development of a computer proficiency 
exam. 

Changing Environments, University Responses 

The sweep of digital technologies and the transformation to a knowledge-based economy have 
created a robust demand for workers highly skilled in the use of information technology. The 
demand for workers who can create, apply and use information technology goes beyond the IT 
industry, cutting across manufacturing and services, transportation, health care, education and 
government. 

In order to meet the new demands resulting from the rapid rise in technology use over the 
past decade, institutions of higher education have endeavored to identify what basic 
competencies students need when they graduate. Computer proficiency at the University of 
Texas at Arlington was officially defined in 1997. UTA, upon the 1997 recommendation of the 
Southern Association of Colleges and Schools (SACS) Reaffirmation Committee, adopted a 
computer proficiency requirement for all undergraduate students. Thel997 SACS Reaffirmation 
Committee Report categorically stated that basic computer use must be appropriate to the 
discipline and that it be clearly and explicitly demonstrated (SACS Criteria for Accreditation, 
Section 4.2.2). 
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UTA concurred that computer proficiency is essential both within the university and in 
the changing job market. The requirement would meet two objectives. First, it would ensure that 
professors could assume that students in their courses were proficient by the end of their 
freshman year. Second, that the university would be graduating students who had the necessary 
skills to succeed in their careers. 

A computer proficiency requirement allows Instructors to focus on the content of their 
courses, rather than training students in the use of technology. Instructors increasingly require 
that students use technology in their coursework. Many Instructors require that students use e- 
mail to communicate, participate in discussion boards, access on-line databases, use PowerPoint 
for presentations, and use basic productivity tools such as word processing. A National Center 
for Educational Statistics study of the use telecommunications faculty by postsecondary 
instructional faculty and staff in Fall 1998 found that 69 percent of full-time and 46 percent of 
part-time faculty used e-mail to communicate with students and about one-third of both full- and 
part-time faculty used course specific web sites (NCES, 2002). 

Furthermore, a degree from an institution of higher education has traditionally indicated 
to employers that students had a basic skill level in core areas. A college degree, regardless of 
the discipline, communicates to employers that graduates have received a certain level of 
education that will aid them in their careers. Math and English proficiency have long been 
requirements for a university education because they are basic to success in those employment 
areas that require a college degree. Over the past decade, computer proficiency has emerged as 
an additional area that is consistently valued by employers. 

The job market has seen a dramatic increase in the demand for employees capable of 
incorporating technology and information management skills in their jobs. Although the 
technology skills of students currently graduating from high schools and universities surpass the 
skills of students graduating just ten years ago, employers are not yet able, or not yet willing, to 
assume that candidates know how to use technology and manage information. (Improving the 
Responsiveness Between Industry and Higher Education. 1999. 

http://www.itaa.org/workforce/studies/response.htm.) Requiring computer proficiency for 
graduation, like requiring Math and English proficiency, provides employers with critical 
information about the skills of college graduates. 




3 



However, specifying demonstration of computer proficiency is complicated by the fact that 
base skill levels differ drastically. Many traditional students have been using computers since 
elementary school. Non-traditional students, particularly those who were not exposed to 
computers in school or the workplace before coming to campus, require more extensive training. 
Thus, requiring all students to take a course aimed at basic proficiency would likely prove 
frustrating for students who were already at the minimum skill level. However, requiring a more 
advanced computer course would not be logical for many majors. 

The implementation of the computer proficiency requirement at UTA addressed these 
varying levels through the development of a Computer Proficiency Test that would allow 
advanced students to demonstrate proficiency without taking a course. Currently, students have 
two options for fulfilling the computer proficiency requirement: 

1. ) The student may opt to take the course designated within their major department as 

that which satisfies the computer proficiency requirement. 

2. ) The student may opt to take the Computer Proficiency Exam developed by 

Assessment Services at the University of Texas at Arlington. 

As the first option indicates, UTA decided that the decision as to what course would satisfy 
the computer proficiency requirement ought to be left with the department. This was done for 
two reasons. It was decided that each discipline had specific knowledge about the unique 
computer requirements of their field. For some departments, these requirements were quite 
stringent and thus the course required to meet the computer proficiency requirement was part of 
the students’ core curriculum. In this case, the student would never be a candidate for the 
Computer Proficiency Test. Other departments determined that the computer proficiency level 
demanded by their discipline was more broadly based - there were no specific software 
programs that were essential for success. Thus students could satisfy the requirement with a 
general course. Those students who already had the skills to be taught in that course would be 
candidates for the Computer Proficiency Test. 

The second objective was met by leaving the decision in the department was that 
departments became focused on assessing the computer skills needed for students they were 
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graduating. It was anticipated that this would have a positive affect on the use of technology on 
campus. 

Whether or not increased technology use is the result of the computer proficiency 
requirement, or some other intervening variable, is unclear. However, a survey of department 
chairs (N=25) conducted in Fall 2002 indicates that many do believe that technology use in the 
classroom has increased since the requirement was implemented. When asked, do you believe 
that the level of technology use in the classroom has increased since the computer proficiency 
requirement implementation in 1999? 29 percent responded “definitely yes,” 42 percent 
responded “probably yes,” 21 percent responded “unsure,” 8 percent responded “probably no” 
and no respondents indicated that they thought, “definitely no,” the level of technology had not 
increased. 

UTA Computer Proficiency Test Development 

Because the decision of what course would satisfy the computer proficiency requirement was left 
with the department, in some cases the designated course was part of the student’s core 
curriculum. These students would not qualify to take the test. Thus, during the initial 
development of the Computer Proficiency Test, it was difficult to estimate how many students it 
would be necessary to test. If the number of students were expected to be extremely high, such as 
the volume of students who take Math Placement Exams, then the volume of students expected 
to take the test would have to play some role in deciding how the test was developed and 
administered. 

Certainly, a multiple-choice test would be a more efficient means to test a high volume of 
students. However, concerns regarding the construct validity of such a test immediately surfaced. 
It seemed unlikely that a multiple-choice test could accurately measure computer proficiency. 

For example, if the ability to copy and paste text were determined to be an indicator of 
proficiency, a multiple-choice test might ask the student to choose from a list of items that were 
representative of the steps necessary to complete the task. However, there are, at minimum, three 
different ways to accomplish this task; all of them valid. This was significant concern and, when 
balanced with the possibility that there may be a high number of students taking the test, it was 
decided that the measurement instrument itself was of primary importance. 
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Therefore, the test developed at UTA is a performance-based, or authentic, assessment. 
Grounded in William Spady’s philosophy of Outcomes Based Education (OBE), authentic 
assessment is a performance opportunity for students requiring them to demonstrate outcomes 
that result in either solutions to meaningful real-life problems or significant purposeful products. 
Thus, an outcome is a culminating demonstration of the entire range of learning experiences and 
capabilities, including higher order thinking skills (Grandgenett, 1999). 

Authentic assessment avoids the artificial division of knowledge into discrete items in 
favor of measuring behavior as a whole (Yen, 1993, in Hoepfl, 2003). The increased use of 
authentic assessment is grounded in broader reform efforts aimed at overall curriculum change. 
According to Marie Hoepfl, these reform efforts stem from three sources: a backlash against the 
pressure for accountability through standardized testing, the expansion of cognitive science, and 
concern from the business community that schools are not adequately preparing youth for 
today’s workplace (Hoepfl, 2003, p.49). 

In order to develop an authentic assessment of computer proficiency, UTA began with 
the identification of five competency areas with associated learning outcomes, general indicators, 
and performance measurements (see Appendix 1). Once the competencies were identified, they 
were further broken down into specific tasks that, when combined, would produce a cohesive 
product or outcome. 

The Computer Proficiency Test is a modified combination of product-oriented and 
portfolio assessment. A product-oriented assessment requires students to produce tangible 
products, e.g. a visual display. A portfolio assessment is a collection of samples of student work 
(Grandgenett, 1999). Rather than measuring a student’s ability to memorize keystrokes and pull- 
down menus, the test was designed to measure a students’ ability to use the tools available in 
order to perform meaningful, real-life tasks. Because the Computer Proficiency Test requires that 
the student perform a task in a proctored environment, it is, of course, an abbreviated version of a 
portfolio. 

The test is divided into five components, each measuring one of the five competency 
areas: Word Processing, Spreadsheets, Internet Research and Evaluation, E-mail, and On-line 
Library Usage. Beyond knowing how to use these tools, students are required to analyze their 
utility. Additionally, there are several built-in components that are designed to expose students to 
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more advanced uses of the technology without requiring that they are proficient in these more 
advanced tasks. 



Word Processing 

o The student is asked to modify a document, according to the instructions, using a word 
processing program. 

o Desktop management is also incorporated in this component. 



Spreadsheets 

o The student is asked to create a basic spreadsheet and graph, 
o In addition the test exposes them to more powerful uses, such as using pre-built 
calculators to answer questions. 

Internet Research and Evaluation 
o The student is asked to: 

o search for information on a given topic, 
o choose a web site, 

o evaluate the web sites using an internet evaluation form, prepare a bibliography for 
the sites, 

o use the UTA web page and gather information. 

E-mail 

o The student is asked to use their UTA e-mail account, via remote access, and send an e- 
mail with and attachment from a disk, to the Test Administrator. 

On-line Library Usage 

o use the library web site to research a topic related to their field of study 
o print full-text articles, 

o determine if a book of their choice is currently available in the library, 
o and e-mail database search records to the test administrator. 

Consistent with the philosophy of OBE, students are well prepared for the test. Assessment 
Services provides a tutorial for each component. The tutorials consist of a series of screen 
captures taken at each step of the tasks which students are required to perform, accompanied by a 
text box describing the action. The tutorials are available as hard copies or on CD-ROM. 

Test Administration 

Large-scale administration of an authentic assessment presents unique challenges. Hoepfl 
points to Roeber’s division of these challenges into two categories: practical and technical. 
Technical challenges include invalidity, fairness, reliability, generalizability, and comparability. 
Practical challenges include multiple purposes, cost, and time (Hoepfl, 2003, p. 54). In the case 
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of computer proficiency, UTA considered an authentic assessment more likely to meet the 
standards of construct and content validity because of the direct relationship between the 
objective and the measurement being used (see Appendix I). However, the issues of reliability, 
generalizability, and comparability proved more difficult. 

While the student is given a comprehensive task, such as the creation of a document, that 
task is broken down for grading purposes. The use of a point system for grading, addressed 
below, was used in order to increase the reliability of the test and remove subjectivity. In order to 
address issues of generalizability, or the ability of the assessment to assess the broader skill set, 
the test incorporates higher order thinking skills and problem solving (see Appendix I). Data is 
currently being collected in order to measure the comparability, or consistency, of the test. 

The practical concerns of purpose, cost and time have not proved overwhelming at UTA; 
primarily because of the single purpose of the test, a staff position dedicated to the test and a 
reasonably low demand. The Computer Proficiency Test is currently offered twice a month. The 
exam is administered in a proctored computer-testing center. Students are allocated four hours to 
complete the exam. Many students complete the exam in two hours, however there has been no 
correlation between the time taken to complete the exam and passage rate. 

The exam is graded using a point system. Each component is broken down into tasks. For 
example, one portion of the Word Processing test might include: 

Word Processing 

□ Open the Word document titled “test” (1 point) 

□ Change the font size from 10 pt. to 12 pt. (1 point) 

□ Change the font from Times New Roman to Arial (1 point) 

□ Indent the paragraphs (1 point) 

□ Full justify the test. (1 point) 

A student who successfully modifies the document will have performed each of these 
tasks. Other skills are incorporated into each component in order to increase generalizability and 
measure higher order thinking skills. For example, the disk containing the “test.doc” file will 
also contain a spreadsheet file, a PowerPoint file, and an HTML file titled test, thus requiring the 
students to demonstrate their ability to distinguish between file types. The word processing test, 
as part of a measure of their knowledge of how to make a bulleted list, requires that they list 
advantages and disadvantages of using Internet sources for research. Additionally, they are 
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required to save their results in a folder with their name on the disk in order to demonstrate file 
management skills. 

Each component is graded separately, with 80% passing. A student who fails one 
component of the test is not required to re-take the entire exam - only the portion they failed. 
Grading the test is time consuming. An experienced grader will spend, on average, 30 minutes 
grading one exam. 

Since fall 1999, 83 Computer Proficiency Tests have been administered: 



2000 


5 


2001 


18 


2002 


50 


2003 (3 months) 


8 (7 registered for April) 



Each component of the test is graded separately. The majority of students pass the Word 
Processing, Internet, E-mail, and Library components on the first try. Students who fail Word 
Processing tend to fail the entire exam. Approximately 10 percent of examinees failed the 
Spreadsheet component on their first try. A student who fails a component of the test may re-take 
that component six weeks later. A student may re-take a slightly modified version of a 
component once. Those students who fail a component twice are advised to register for a full 
course. 

Other Higher Education Models 

Other institutions of higher education have faced similar mandates. A Google search for 
“computer proficiency requirement,” limited to institutions of higher education, reveals 417 hits. 
Certainly, this is a less than scientific method. However, a cursory examination of these results 
indicates that many colleges and universities have implemented a computer proficiency 
requirement, with several offering the option to take a test as a means of fulfilling the 
requirement. It also appears that many departments have instituted their own computer 
proficiency requirements. 
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Of those institutions of higher education that appear to have instituted a requirement. The 
University of Missouri-Columbia, Centenary College, and Southeastern Oklahoma State 
University, University of the Sciences in Philadelphia, Belmont University, and Northeastern 
State University, all require computer proficiency, which can be satisfied through course 
completion or an exam. A more comprehensive and methodological search would likely find 
many more. The exam at Southeastern, like the UTA exam, includes an authentic assessment of 
word processing and Internet research skills. In addition, their exam incorporates software, 
hardware, and ethics. 

Virginia Commonwealth University uses a somewhat different model. Their web page 

indicates that VCU Students may be required to take a computer proficiency test in order to 

graduate, or as an entrance requirement to one of VCU's graduate programs. They list four 

groups that are required to take the exam: those graduating from the College of Humanities and 

Sciences, all incoming Freshmen, incoming MBA Students, and incoming Pharm. D. Students. 

The exam is part of a more comprehensive system, SmartForce: 

“SmartForce also offers Tutorials (or smartcourses) to all faculty, staff and students who 
are interested in learning how to use various software programs and products including 
Microsoft, Macromedia, Lotus, Java, etc. Students can also take the tutorials in 
preparation for the Assessments, (http://www.vcu.edu/cte/smartforce/)” 

UTA was particularly interested in the passage rate of students at VCU. Information on their web 
page indicates that: 

• Students whose scores ranged from 80-100 were classified as Highly Competent. 

• Students whose scores ranged from 60-79 were classified as Competent. 

• Students whose scores fell below 60 were classified as Less than Competent. 



= 

1 


n = # who 
completed 
SmartForce 
assessment 


Highly 

Competent 


Competent 


Less than 
Competent 


General Knowledge and 
Operating Systems 


194 


121-62% 


73-38% 


0 

^ 1 


Word Processing 


197 


108-55% 1 


89-45% 


0 


Using the Web | 


166 


69-42% 


97-58% 


0 


Using E-mail 


186 


82-44% 


104-56% i 


0 



(http://www.vcu.edU/cte/smartforce/faqs.html#faring) 



In considering a system of training and assessment similar to that used by VCU, UTA 
determined that the nature of the exams, which are multiple-choice, would not meet the 
University’s needs. There is a trade-off between authentic assessment and volume of students 
that can be accommodated. However, if the volume of students taking the exam were higher, or 
demand for tutorials on campus greater, it would certainly be considered. 

In 1999 The Virginia Foundation for Independent Colleges (VFIC) developed the 
Tek.Xam. The test was originally requested by the area business community in Virginia as a 
certification of technology proficiency in the hopes that it could be used similar to a Microsoft 
Certification. It was certified by several East coast corporations when UTA offered the Tek.Xam 
in Spring 2000, however no students registered for the exam. Since that time, it appears that the 
Tek.Xam is now marketed as an exam available to institutions of higher education to test 
computer proficiency of students. More information is available at 

http://www.tekxam.com/index.htm. The Tek.Xam includes General Computing Concepts, 
Presentations, Spreadsheets, Web Design, and Word Processing. A more specific outline of the 
test can be found at http://www.tekxam.com/StudyGuide/Guide.html. 

Expanded Computer Proficiency Testing 

A modified version of the Computer Proficiency Test was prepared for the Department of 
Human Resources at UTA. The test was intended to test current employees as part of their 
Employee Development Plan. Three levels were established and two components, Microsoft 
PowerPoint and Microsoft Access, were added. 

Level I - Basic Word Processing, E-mail, and UTA web site use. 

Level II - Word Processing n. Spreadsheets, E-mail, Advanced Internet. 

Level III - Advanced Word Processing, Spreadsheets, E-mail, Advanced Internet, Access, 
PowerPoint. If employee has taken Level II they will only be required to take the Advanced 
Word Processing, PowerPoint and Access Portions of the test. 

The structure of this exam could serve as a model for more department-specific exams. 
An exam can be developed for any number of components, including the original five. For 
instance, a statistical software component could be added to meet the needs of the Psychology 
department. However, this model ought to be approached with some caution for two reasons. 
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One, it may lead campus assessment offices down a slippery slope towards the development of 
any number of specific computer proficiency exams. Second, many of these software packages 
ought to be taught within their discipline - a fact well recognized by Department Chairs. In the 
2002 UTA Faculty Survey, 21 percent of Department Chairs indicated “definitely yes” when 
asked if they thought the standard of computer proficiency should differ by discipline, with 45 
percent indicating “probably yes.” Seventeen percent indicated they were “not sure” and 17 
percent indicated “probably no.” No one indicated “definitely no.” 

However, some departments at UTA have indicated that they would like to use the 
Computer Proficiency Test as a benchmark exam that would allow them to assume that students 
in their courses had a minimum level of proficiency so that they could move on to more 
advanced task. This is probably reflective of some overlap with general skills. The Fall 2002 
survey asked: The Computer Proficiency Exam includes five components: word processing, 
spreadsheets, Internet research productivity, email, and use of on-line catalogs of the library. 
How consistent are these components with what is included in your department-designated 
computer proficiency course? Fifty-six percent of respondents indicated “very consistent,” 32 
percent indicated “somewhat consistent,” and no one indicated “not at all consistent.” 

Student Survey 

A student survey was administered during the Fall 2002 and Spring 2003 semesters (N=22). Our 
first concern was with how well students felt they were prepared for the test and whether it 
contained the content they expected to be tested over. 



How did you study for the test? 
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If you used the Assessment Services tutorial, was it helpful? 




Our next concern was how students felt about the difficult of the test and the test format. 



Did you feel that the test measured your proficiency level? 




How would you rate the difficulty of the test? 
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Would you rather take a multiple choice test than the test you took today? 




Conclusion 

When UTA first began development of the Computer Proficiency Test in 1999, there were 
virtually no models available and it was difficult to obtain information about computer 
proficiency requirements in higher education. In the four years since then, as evidenced by the 
cursory examination previously referred to, there has been an increase in the number of colleges 
and universities requiring that students demonstrate computer proficiency prior to graduation. It 
is somewhat reassuring that many universities, apparently operating independent of one another, 
have developed similar programs. However, there still remains a lack of cohesive literature in the 
area of computer proficiency and its assessment. Further research is needed regarding how such 
policies are implemented, their effectiveness, and the reliability of measurement of this unique 
skill set. 
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Appendix I 

University of Texas at Arlington 
Student Computer Proficiency Outcomes 



Learning 

Outcomes 


General 

Indicators 


Performance Criteria 


Performance Measurement 


Word 

Processing 


Student can create, 
modify, save, and 
print documents. 


1. Create file 

2. Cut and paste text 

3. Copy text 

4. Save file as/to 

5. Create a table 

6. Underline text 

7. Bold face text 

8. Italicize text 

9. Justify text 

10. Change font size 

11. Change font face 

12. Double space the 
document 

13. Change margins 

14. Create a bulleted 
list 

15. Spell checks the 
document. 


Given a document, the student will 
successfully alter the document to 
administrator specifications, spell 
check the document, save the 
document to a floppy disk, and print 
the document. 


Desktop/file 

management 


Student can use 
peripherals, multi- 
task, and identify 
operating systems 
and software. 


1. Use the keyboard 
and mouse. 

2. Minimize windows 

3. Create file folders 

4. Copy and paste 
from one folder to 
another folder, 
desktop, and disk. 

5. Differentiate 
between software 
programs, (identify 
file type by “tag” 
(.ppt, .doc, etc.) 


The student will use desktop and 
file management skills to complete 
other measurement tasks. 


Spreadsheets 


Student uses 
spreadsheets to 
manage and 
interpret 
information. 
Student 

understands the 
capabilities of 


1 . enter data 

2. format cells 

3. create chart 

4. modify chart 

5. sort data 

6. use wizards 

7. use pre-formatted 
worksheets to 


Given a list of data, the student will 
enter the data, modify the 
spreadsheet to administrator 
specifications, and create a chart to 
administrator specifications. 

Given a series of questions the 
student will use a pre-formatted 
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spreadsheet 

programs. 


determine 
relationships 
between numbers 


worksheet to determine the answers. 


On-line 

Library 

Usage 


Student 
successfully 
performs catalog 
and database 
searches for 
citations on a 
given topic. 


1. Identify 
appropriate 
database for 
subject matter 

2. Log on to the 
database from an 
off-site location 

3. Limit searches 


Given a subject, the student will 
identify resources in the library, 
where they are located, and identify 
them as available or not available. 

Given a research topic, the student 
is able to identify, print, and send 
by e-mail, 2 Full Text sources of 
information using appropriate an 
academic database. 


Internet 

Search 

Productivity 


Student 
successfully 
performs an 
Internet search for 
information on 
given topic and is 
able to create a 
bibliography. 


1. Determine 
appropriate search 
engine 

2. narrow search 

3. use URLs 

4. evaluate quality of 
site 

5. copy and paste 
from Internet into 
a word processing 
program 

6. create a 
bibliography for 
web pages using a 
reputable style 
guide 

7. Use the UTA web 
page to find 
information 
regarding their 
activities on 
campus. 


Given a research topic, students will 
identify 5 relevant URLs, evaluate 
the web pages, and create a 
bibliography using the proper 
format for their chosen style. 

Given a question, the student is able 
to use the UTA web page to find 
information. 


E-mail 


Student is able to 
communicate via 
e-mail. 


1. Create e-mail 

2. Attach file 

3. Carbon copy a 
message 


The student is able to log in to their 
UTA e-mail account, create a 
message with an attached file, and 
send the message to two recipients. 
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